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ABSTRACT 


For  the  Department  of  Energy  (DOE)  Knowledge  Base  to  support  activities  for  monitoring  nuclear  explosions 
consistent  with  eventual  verification  activities  under  the  Comprehensive  Nuclear-Test-Ban  Treaty  (CTBT),  aprocess  is 
defined  to  ensure  the  integrity  and  utility  of  research  results  during  the  migration  into  information  products  for  use  in 
operational  monitoring  systems. 

The  process  of  validating,  verifying,  and  managing  the  information  products  ensures  deliveries  of  high-quality 
Knowledge  Base  releases  to  the  United  States  National  Data  Center  (USNDC).  These  activities  are  critical  to  the 
successful  integration  of  scientific  research  to  support  operational  monitoring  systems  at  the  USNDC.  A  by-product  of 
this  process  is  that  data  sets,  or  components  of  information  products,  that  have  undergone  the  validation  and 
verification  process  may  be  distributed  as  operational  calibration  products  to  the  International  Data  Centre.  All 
contributors  to  information  products,  whether  DOE-funded  or  not,  will  benefit  from  transparency  of  the  integration 
process  to  effect  successful  participation  in  the  process. 

As  an  information  product  passes  through  the  steps  necessary  to  become  part  of  a  delivery  to  the  USNDC,  domain 
experts,  including  the  end-users,  will  provide  validation  —  a  determination  of  relevance  and  scientific  quality.  The 
integration  process  continues  with  verification  —  an  assessment  of  completeness  and  correctness,  provided  by  the 
Knowledge  Base  integrator,  the  information  product  coordinator,  and  the  contributing  organization.  The  information 
products  and  their  constituent  data  sets  are  systematically  tracked  through  the  integration  portion  of  their  life  cycle 
(Moore,  et  al,  2000;  Carr  et  al,  2000). 

Finally,  the  proposed  delivery  of  the  Knowledge  Base  and  its  constituent  information  products  is  reviewed  by  an 
Integration  Board.  The  integration  process  is  presented  in  this  paper,  with  details  described  in  Moore  et  al.,  (2000). 
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OBJECTIVE 

A  well-defined  process  ensures  the  integrity  and  utility  of  research  results  during  the  migration  into  information 
products  for  use  in  the  DOE  Knowledge  Base  to  support  activities  for  monitoring  nuclear  explosions  consistent  with 
eventual  verification  activities  under  the  Comprehensive  Nuclear-Test-Ban  Treaty  (CTBT).  The  process  of  validating, 
verifying,  and  managing  the  information  products  ensures  high-quality  deliveries  to  the  United  States  National  Data 
Center  (USNDC). 

RESEARCH  ACCOMPLISHMENTS 

The  process  for  migrating  research  results  into  information  products  for  use  in  the  DOE  Knowledge  Base  has  been 
defined  and  implemented.  This  process  and  its  implementation  ensure  that  the  delivery  of  a  DOE  Knowledge  Base 
comprises  verified,  validated,  and  well-managed  information  products.  If  any  process  is  to  retain  its  effectiveness,  it 
must  be  refined  over  the  course  of  its  use;  thus,  this  process  is  also  undergoing  refinement.  This  paper  describes  the 
integration  process  as  documented  in  Moore  et  al.  (2000)  with  refinements  developed  during  its  most  recent 


implementation.  This  paper  serves  the  purpose  of  offering  transparency  of  the  integration  process  to  facilitate  effective 
participation  in  the  process  by  all  contributors  to  information  products. 

Motivation  and  benefits  of  the  integration  process 

For  the  DOE  Knowledge  Base  to  be  useful  as  a  tool  for  improving  U.S.  monitoring  capabilities,  the  contents  of  the 
Knowledge  Base  must  be  subjected  to  a  well-defined  set  of  procedures  to  ensure  integrity  and  relevance.  The 
integration  process  addresses  the  following  issues: 

1.  What  do  end-users  want  and  need? 

2.  How  do  researchers  collaborate  to  address  those  needs? 

3.  How  is  research  integrated  into  a  useful  and  reliable  product? 

These  questions  drove  the  development  of  the  integration  process  and  are  answered  explicitly  at  the  end  of  this  paper 
after  the  process  is  described. 

In  addition  to  ensuring  integrity,  relevance,  and  usefulness  of  the  information  products,  the  integration  process  provides 
additional  benefits: 

•  Evidence  that  the  process  was  followed.  All  steps  in  the  process  are  logged. 

•  Direction  for  interactions.  Procedural  guidance  provided  by  the  process  ensures  effective,  purposeful 
interactions. 

•  Programmatic  benefits.  The  procedure  provides  a  basis  for  scheduling  and  offers  periodic  status  reports. 

Description  of  the  process 

Figure  1  summarizes  the  state  changes  for  information  products  (IP). 


Note:  Underlined  words  are  states 


Fig.  1.  State  transition  diagram  for  Information  Product  integration  into  the  Knowledge  Base. 


The  following  nomenclature  is  used  for  the  integration  process  and  Knowledge  Base  releases.  An  information  product 
comprises 

•  one  or  more  conceptually  related  data  sets 

•  metadata  corresponding  to  each  data  set 

•  an  information  product  release  document  (described  in  state  1  below) 

•  any  associated  models,  interface  tools,  or  visualization  tools. 

A  data  set  is  a  collection  of  data  usually  all  of  the  same  kind;  examples  include  event  data,  contextual  data,  or 
parametric  grid  data.  Metadata  describes  the  content,  source,  quality  and  condition  of  a  data  set. 

The  integration  process  defines  states  through  which  an  information  product  passes  as  it  migrates  from  the  research 
environment  into  the  Knowledge  Base.  Though  the  integration  process  is  a  joint  effort,  Sandia  National  Laboratories 
(SNL)  holds  the  primary  responsibility  for  integrating  information  products  into  the  Knowledge  Base,  so  the  states  in 
the  diagram  are  defined  relative  to  SNL’s  role.  As  the  Knowledge  Base  integrator,  SNL  systematically  tracks  each  of 
the  states  for  each  of  the  information  products  offered  for  inclusion  in  the  Knowledge  Base. 

1.  The  initial  state  of  the  process  is  Draft  Information  Product  (IP)  Release  Document  Received.  A  release  document 
is  developed  by  a  contributor  (a  research  organization,  typically)  and  contains  at  a  minimum: 

•  information  product  identification 

•  content  description 

•  changes  since  prior  versions 

•  installation  instructions. 

When  the  draft  version  of  the  release  document  is  received,  it  indicates  that  the  contributor  intends  to  submit  an 
information  product  for  inclusion  in  the  Knowledge  Base,  so  the  Knowledge  Base  integrator  begins  tracking  that 
proposed  information  product.  The  Knowledge  Base  integrator  also  identifies  two  points  of  contact  for  this  specific 
information  product: 

•  An  information  product  coordinator  who  represents  the  contributing  organization. 

•  An  information  product  integrator  ( representing  the  Knowledge  Base  integrator  organization)  who  will  work 
with  this  particular  information  product.  This  paper  will  continue  to  refer  to  the  Knowledge  Base  integrator 
for  clarity  in  process  description,  but  it  is  a  specific  person,  the  information  product  integrator,  who  performs 
the  integration  and  tracking  for  a  particular  information  product. 

The  Knowledge  Base  integrator  reviews  the  draft  release  document  for  completeness.  If  unsatisfactory,  the  draft  release 
document  is  returned  to  the  contributor  with  feedback.  If  the  draft  release  document  is  found  to  be  satisfactory,  the 
information  product  is  promoted  to  the  next  state. 

2.  In  this  state,  Draft  Information  Product  Document  Verified,  the  proposed  information  product  represented  by  the 
draft  release  document  is  subjected  to  technical  review  via  the  Knowledge  Base  Advisory  Group  for  that 
information  product.  The  Advisory  Group  comprises 

•  domain  experts  from  peer  organizations  and 

•  one  or  more  representatives  from  the  end-user  organization. 

The  Advisory  Group  provides  validation,  a  determination  of  relevance  and  scientific  quality,  of  the  information  product 
proposed  for  inclusion  in  the  Knowledge  Base.  The  assessment  is  based  on  whether  or  not  the  proposed  information 
product  supports  the  defined  monitoring  goals.  Written  assessment  and  recommendations  are  provided  to  the 
contributor  and  to  the  Knowledge  Base  integrator. 

If  the  Advisory  Group  deems  the  information  product  (represented  by  its  draft  release  document)  unsuitable,  then  the 
draft  release  document  is  returned  to  the  contributor  with  feedback.  If  satisfactory,  the  information  product  is  promoted 
to  the  next  state. 

3.  At  this  point.  Draft  Information  Product  Release  Document  Validated,  the  contributor  prepares  the  delivery  of  the 
information  product  by 

•  finalizing  the  release  document, 

•  synthesizing  the  research  data, 

•  completing  the  associated  metadata,  and 


completing  associated  interface  tools. 


Meanwhile,  the  Knowledge  Base  integrator  (SNL)  uses  the  validated  draft  release  document  as  a  basis  for  integrating 
the  information  product  into  the  Knowledge  Base.  For  example,  the  release  document  provides  information  about 
quantity  and  format  of  the  forthcoming  data,  which  helps  with  system  configuration,  both  in  the  integration  test  bed  and 
at  the  end-user  site.  Also,  the  release  document  provides  information  about  testing  and  verifying  the  forthcoming 
information  product,  which  aids  in  the  verification  process.  It  provides  information  about  associated  process  tools  and 
interface  tools,  which  helps  coordinate  tool  development  and  testing. 

4.  The  integration  process  continues  when  the  contributor  delivers  the  comprehensive  information  product  for 
integration;  SNL  logs  this  state  as  Information  Product  Delivery  Received. 

The  information  product  is  considered  received  when  all  data  sets  comprising  the  information  product  have  been 
received,  all  metadata  received,  the  final  release  document  received,  and  any  interface  tools  for  process  or  visualization 
have  been  received. 

The  Knowledge  Base  integrator  performs  extensive  verification  of  the  information  product  for  completeness, 
consistency,  and  correctness,  based  on  the  preparation  using  the  release  document.  The  integrator  tracks  each  data  set 
contained  in  the  information  product  and  its  associated  metadata  for  both  receipt  and  verification.  The  integrator  may 
return  the  information  product  to  the  contributor  with  feedback  if  the  information  product  or  one  or  more  of  its  data  sets 
are  found  to  be  unacceptable,  incomplete,  or  missing.  If  acceptable,  the  information  product  is  promoted  to  the  next 
state. 

5.  This  state,  Information  Product  Delivery  Verified,  indicates  the  information  product  is  ready  to  be  integrated  into  a 
Knowledge  Base  release. 

At  this  point,  several  verified  information  products  are  integrated  into  a  potential  Knowledge  Base  release.  The 
combined  release  is  verified  at  a  system  level,  appropriate  classification  issues  are  addressed,  and  the  Knowledge  Base 
integrator  develops  a  Knowledge  Base  release  document.  The  Knowledge  Base  integrator  synthesizes  the  following  to 
complete  the  integration  into  the  potential  Knowledge  Base  release: 

•  the  comprehensive  potential  release  and  its  release  document, 

•  information  product  release  documents, 

•  Advisory  Group  recommendations,  and 

•  tracking  reports  demonstrating  the  verification,  validation,  and  management  of  the  information  products. 

6.  With  verification  activities  completed,  the  state  now  becomes  Integrated  Knowledge  Base  Release  Verified,  which 
means  the  proposed  Knowledge  Base  release  and  its  supporting  documentation  are  ready  to  be  presented  to  the 
Integration  Board  for  approval. 

In  addition  to  the  potential  release  and  its  supporting  documentation  presented  to  the  Integration  Board,  contributors  are 
encouraged  to  present  an  overview  of  their  respective  information  products  during  the  Integration  Board  meeting. 

The  purpose  of  the  Integration  Board  is  to  provide  high-level  approval  that  the  information  products  have  been  verified, 
validated,  and  properly  integrated  based  on  recommendations  by  the  Knowledge  Base  Advisory  Groups,  which  have 
accepted  the  information  products  after  technical  review.  The  Integration  Board  comprises,  at  a  minimum,  a  program 
leader  from  each  contributing  organization,  the  DOE  sponsor,  and  a  technical  representative  from  the  end-users’  site. 
Specifically,  the  purpose  of  the  Integration  Board  is  to 

•  review  the  contents  of  the  potential  Knowledge  Base  release 

•  be  advised  by  the  Knowledge  Base  Advisory  Groups’  reports 

•  be  informed  by  the  information  product  status  reports  from  the  tracking  tool  used  in  the  integration  process 

•  accept  or  deny  all  or  part  of  the  potential  Knowledge  Base  rele  ase. 

7.  If  the  Integration  Board  approves  the  potential  release,  then  the  state  becomes  Integrated  Knowledge  Base  Release 
Approved,  and  the  approved  Knowledge  Base  release  is  delivered  to  the  USNDC.  Integration  of  specific 


information  products  into  a  Knowledge  Base  release  does  not  guarantee  acceptance  by  the  USNDC,  which 
exercises  its  own  review  process  after  receipt  of  a  release. 


This  integration  process  can  also  be  summarized  in  terms  of  data  flow,  as  shown  in  Figure  2. 
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Figure  2.  Basic  data  flow  for  Knowledge  Base  Development  and  Integration 
(shown  relative  to  parallel  user  evaluation  process). 


How  the  process  addresses  the  original  issues 

The  integration  process  was  developed  to  address  three  issues  mentioned  early  in  this  paper.  Those  issues  aie explicitly 
addressed  here. 

What  do  end-users  want  and  need? 

The  integration  process  includes  end-users  early  on  with  their  participation  in  the  Advisory  Group  review  of  the  draft 
release  document  for  validity  or  relevance  of  a  proposed  information  product.  The  needs  of  end-users  are  addressed 
even  earlier  by  the  contributing  organizations  with  their  interactions  with  end-users.  These  interactions,  while  outside 
the  scope  of  the  integration  process,  form  the  basis  for  the  research  and  any  resulting  information  products.  Further,  the 
end-users  are  represented  on  the  Integration  Board  to  ensure  their  needs  are  met. 


How  do  researchers  collaborate  to  address  those  needs? 

The  integration  process  provides  guidance  in  collaborating  by  establishing  the  roles  of  information  product  integrator 
and  information  product  coordinator.  These  two  roles  must  work  closely  together  using  a  well-defined  process  to 
ensure  successful  integration  of  scientific  research  results  into  a  useful  information  product.  In  addition,  the  integration 
process  provides  evidence,  or  documentation,  of  the  collaborations  to  support  the  usefulness  and  reliability  of  the 
information  products. 

How  is  research  integrated  into  a  useful  and  reliable  product? 

One  of  the  original  and  enduring  requirements  in  developing  a  process  to  integrate  research  results  into  information 
products  for  use  within  the  Knowledge  Base  is  that  the  process  may  not  prescribe  the  type  or  amount  of  data  developed 
by  the  researchers.  That  is,  the  process  must  integrate  a  wide  range  of  data  types,  data  complexity,  and  data  volume. 
The  purpose  for  this  requirement  is  to  accommodate,  not  constrain,  the  researchers  in  their  work  so  that  the  full  utility 
of  their  research  is  maintained  during  its  integration  into  the  Knowledge  Base. 

CONCLUSIONS  AND  RECOMMENDATIONS 

The  process  of  validating,  verifying,  and  managing  the  information  products  is  critical  to  the  successful  integration  of 
scientific  research  to  support  operational  monitoring  systems  at  the  USNDC.  We  have  defined  and  implemented  a 
successful  process  to  date. 

Continued  success  of  the  integration  process  relies  on  the  following: 

•  The  integration  process  is  rigorous  and  well-defined. 

•  All  contributors,  whether  DOE-funded  or  not,  are  knowledgeable  about  the  process. 

•  The  process  continues  to  be  refined  as  USNDC  requirements  emerge  and  as  various  types  of  research  evolve. 

•  We  get  and  incorporate  feedback  from  the  end-users  concerning  the  usefulness  of  the  format  and 
functionality. 

•  We  get  and  incorporate  feedback  from  the  end-users  regarding  the  validity  of  the  content  and  functionality. 

Near-term  work  on  the  integration  process  includes  the  following: 

•  Evolution  of  process  tools.  The  tracking  tool  will  be  modified  to  incorporate  refinements  to  the  process  and 
enhanced  to  generate  various  types  of  reports  such  as  tracking  by  information  product  or  tracking  by 
contributor. 

•  Refinements  of  process.  As  the  number,  complexity,  and  type  of  information  products  expand,  the  process 
will  need  to  be  refined  to  accommodate  the  growth.  For  example,  as  research  interface  tools  are  developed 
and  included  with  information  products,  they  will  be  integrated  with  equally  rigorous  and  well-defined 
processes  to  ensure  integrity  and  utility. 
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